A physical environment may be mapped by a computer using various sensors to detect locations and/or movement of objects and other features in the environment. However, some environments may be dynamic and include features that move relative to one another. In such environments, applications such as gesture recognition and real-world visual augmentation may be affected by the movement of the real-world features.